{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# process\n",
    "\n",
    "## 介绍\n",
    "提供常用的因子处理操作，如去极值，中性化等"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## standardize\n",
    "- ` jaqs_fxdayu.research.signaldigger.process.standardize(factor_df, index_member=None) `\n",
    "\n",
    "**简要描述：**\n",
    "\n",
    "- 横截面z-score标准化\n",
    "\n",
    "**参数:**\n",
    "\n",
    "|字段|必选|类型|说明|\n",
    "|:----    |:---|:----- |-----   |\n",
    "|factor_df |是|pandas.DataFrame |日期为索引,证券品种为columns的二维因子表格|\n",
    "|index_member |否|pandas.DataFrame of bool |是否是指数成分股。日期为索引,证券品种为columns的二维bool值表格,True代表该品种在该日期下属于指数成分股。传入该参数,则进行标准化所纳入的样本只有每期横截面上属于对应指数成分股的股票，默认为空|\n",
    "\n",
    "**返回:**\n",
    "\n",
    "标准化后的因子\n",
    "\n",
    "**示例：**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "import warnings\n",
    "warnings.filterwarnings('ignore')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Dataview loaded successfully.\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style>\n",
       "    .dataframe thead tr:only-child th {\n",
       "        text-align: right;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: left;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th>symbol</th>\n",
       "      <th>000001.SZ</th>\n",
       "      <th>000002.SZ</th>\n",
       "      <th>000008.SZ</th>\n",
       "      <th>000009.SZ</th>\n",
       "      <th>000027.SZ</th>\n",
       "      <th>000039.SZ</th>\n",
       "      <th>000060.SZ</th>\n",
       "      <th>000061.SZ</th>\n",
       "      <th>000063.SZ</th>\n",
       "      <th>000069.SZ</th>\n",
       "      <th>...</th>\n",
       "      <th>601988.SH</th>\n",
       "      <th>601989.SH</th>\n",
       "      <th>601992.SH</th>\n",
       "      <th>601997.SH</th>\n",
       "      <th>601998.SH</th>\n",
       "      <th>603000.SH</th>\n",
       "      <th>603160.SH</th>\n",
       "      <th>603858.SH</th>\n",
       "      <th>603885.SH</th>\n",
       "      <th>603993.SH</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>trade_date</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>20170502</th>\n",
       "      <td>-0.363380</td>\n",
       "      <td>-0.340032</td>\n",
       "      <td>-0.106714</td>\n",
       "      <td>0.152518</td>\n",
       "      <td>-0.266414</td>\n",
       "      <td>0.216918</td>\n",
       "      <td>0.086421</td>\n",
       "      <td>0.857408</td>\n",
       "      <td>-0.411592</td>\n",
       "      <td>-0.343106</td>\n",
       "      <td>...</td>\n",
       "      <td>-0.366394</td>\n",
       "      <td>0.891601</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>-0.361782</td>\n",
       "      <td>0.677455</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>-0.248940</td>\n",
       "      <td>0.131240</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>20170503</th>\n",
       "      <td>-0.364271</td>\n",
       "      <td>-0.341856</td>\n",
       "      <td>-0.107757</td>\n",
       "      <td>0.151190</td>\n",
       "      <td>-0.268283</td>\n",
       "      <td>0.219121</td>\n",
       "      <td>0.083804</td>\n",
       "      <td>0.852450</td>\n",
       "      <td>-0.412694</td>\n",
       "      <td>-0.344699</td>\n",
       "      <td>...</td>\n",
       "      <td>-0.367529</td>\n",
       "      <td>0.879934</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>-0.363002</td>\n",
       "      <td>0.697502</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>-0.248411</td>\n",
       "      <td>0.128307</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>20170504</th>\n",
       "      <td>-0.364991</td>\n",
       "      <td>-0.340861</td>\n",
       "      <td>-0.107070</td>\n",
       "      <td>0.154148</td>\n",
       "      <td>-0.267100</td>\n",
       "      <td>0.213994</td>\n",
       "      <td>0.078180</td>\n",
       "      <td>0.849831</td>\n",
       "      <td>-0.412865</td>\n",
       "      <td>-0.344161</td>\n",
       "      <td>...</td>\n",
       "      <td>-0.367343</td>\n",
       "      <td>0.871015</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>-0.363119</td>\n",
       "      <td>0.674523</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>-0.248024</td>\n",
       "      <td>0.118993</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>20170505</th>\n",
       "      <td>-0.364277</td>\n",
       "      <td>-0.339788</td>\n",
       "      <td>-0.116436</td>\n",
       "      <td>0.142003</td>\n",
       "      <td>-0.266276</td>\n",
       "      <td>0.199128</td>\n",
       "      <td>0.080549</td>\n",
       "      <td>0.857999</td>\n",
       "      <td>-0.412033</td>\n",
       "      <td>-0.343666</td>\n",
       "      <td>...</td>\n",
       "      <td>-0.365914</td>\n",
       "      <td>0.858166</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>-0.362034</td>\n",
       "      <td>0.659895</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>-0.243558</td>\n",
       "      <td>0.114178</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>20170508</th>\n",
       "      <td>-0.360932</td>\n",
       "      <td>-0.337663</td>\n",
       "      <td>-0.121213</td>\n",
       "      <td>0.133428</td>\n",
       "      <td>-0.265375</td>\n",
       "      <td>0.197282</td>\n",
       "      <td>0.087274</td>\n",
       "      <td>0.871560</td>\n",
       "      <td>-0.408468</td>\n",
       "      <td>-0.340375</td>\n",
       "      <td>...</td>\n",
       "      <td>-0.361849</td>\n",
       "      <td>0.824399</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>-0.358094</td>\n",
       "      <td>0.662941</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>-0.242522</td>\n",
       "      <td>0.121454</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>5 rows × 330 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "symbol      000001.SZ  000002.SZ  000008.SZ  000009.SZ  000027.SZ  000039.SZ  \\\n",
       "trade_date                                                                     \n",
       "20170502    -0.363380  -0.340032  -0.106714   0.152518  -0.266414   0.216918   \n",
       "20170503    -0.364271  -0.341856  -0.107757   0.151190  -0.268283   0.219121   \n",
       "20170504    -0.364991  -0.340861  -0.107070   0.154148  -0.267100   0.213994   \n",
       "20170505    -0.364277  -0.339788  -0.116436   0.142003  -0.266276   0.199128   \n",
       "20170508    -0.360932  -0.337663  -0.121213   0.133428  -0.265375   0.197282   \n",
       "\n",
       "symbol      000060.SZ  000061.SZ  000063.SZ  000069.SZ    ...      601988.SH  \\\n",
       "trade_date                                                ...                  \n",
       "20170502     0.086421   0.857408  -0.411592  -0.343106    ...      -0.366394   \n",
       "20170503     0.083804   0.852450  -0.412694  -0.344699    ...      -0.367529   \n",
       "20170504     0.078180   0.849831  -0.412865  -0.344161    ...      -0.367343   \n",
       "20170505     0.080549   0.857999  -0.412033  -0.343666    ...      -0.365914   \n",
       "20170508     0.087274   0.871560  -0.408468  -0.340375    ...      -0.361849   \n",
       "\n",
       "symbol      601989.SH  601992.SH  601997.SH  601998.SH  603000.SH  603160.SH  \\\n",
       "trade_date                                                                     \n",
       "20170502     0.891601        NaN        NaN  -0.361782   0.677455        NaN   \n",
       "20170503     0.879934        NaN        NaN  -0.363002   0.697502        NaN   \n",
       "20170504     0.871015        NaN        NaN  -0.363119   0.674523        NaN   \n",
       "20170505     0.858166        NaN        NaN  -0.362034   0.659895        NaN   \n",
       "20170508     0.824399        NaN        NaN  -0.358094   0.662941        NaN   \n",
       "\n",
       "symbol      603858.SH  603885.SH  603993.SH  \n",
       "trade_date                                   \n",
       "20170502          NaN  -0.248940   0.131240  \n",
       "20170503          NaN  -0.248411   0.128307  \n",
       "20170504          NaN  -0.248024   0.118993  \n",
       "20170505          NaN  -0.243558   0.114178  \n",
       "20170508          NaN  -0.242522   0.121454  \n",
       "\n",
       "[5 rows x 330 columns]"
      ]
     },
     "execution_count": 2,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from jaqs_fxdayu.data import DataView\n",
    "from jaqs_fxdayu.research.signaldigger.process import standardize\n",
    "\n",
    "# 加载dataview数据集\n",
    "dv = DataView()\n",
    "dataview_folder = './data'\n",
    "dv.load_dataview(dataview_folder)\n",
    "\n",
    "# z-score标准化\n",
    "standardize(factor_df = dv.get_ts(\"pe\"), index_member = dv.get_ts(\"index_member\")).head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## winsorize\n",
    "- ` jaqs_fxdayu.research.signaldigger.process.winsorize(factor_df, alpha=0.05, index_member=None) `\n",
    "\n",
    "**简要描述：**\n",
    "\n",
    "- 横截面去极值\n",
    "\n",
    "**参数:**\n",
    "\n",
    "|字段|必选|类型|说明|\n",
    "|:----    |:---|:----- |-----   |\n",
    "|factor_df |是|pandas.DataFrame |日期为索引,证券品种为columns的二维因子表格|\n",
    "|alpha |否|float|去极值的边界，如0.05代表去掉左右两边各2.5%分位的极端值(保留中心部分95%分布的数据)。默认0.05|\n",
    "|index_member |否|pandas.DataFrame of bool |是否是指数成分股。日期为索引,证券品种为columns的二维bool值表格,True代表该品种在该日期下属于指数成分股。传入该参数,则进行去极值所纳入的样本只有每期横截面上属于对应指数成分股的股票，默认为空|\n",
    "\n",
    "**返回:**\n",
    "\n",
    "去极值后的因子\n",
    "\n",
    "**示例：**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style>\n",
       "    .dataframe thead tr:only-child th {\n",
       "        text-align: right;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: left;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th>symbol</th>\n",
       "      <th>000001.SZ</th>\n",
       "      <th>000002.SZ</th>\n",
       "      <th>000008.SZ</th>\n",
       "      <th>000009.SZ</th>\n",
       "      <th>000027.SZ</th>\n",
       "      <th>000039.SZ</th>\n",
       "      <th>000060.SZ</th>\n",
       "      <th>000061.SZ</th>\n",
       "      <th>000063.SZ</th>\n",
       "      <th>000069.SZ</th>\n",
       "      <th>...</th>\n",
       "      <th>601988.SH</th>\n",
       "      <th>601989.SH</th>\n",
       "      <th>601992.SH</th>\n",
       "      <th>601997.SH</th>\n",
       "      <th>601998.SH</th>\n",
       "      <th>603000.SH</th>\n",
       "      <th>603160.SH</th>\n",
       "      <th>603858.SH</th>\n",
       "      <th>603885.SH</th>\n",
       "      <th>603993.SH</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>trade_date</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>20170502</th>\n",
       "      <td>6.7925</td>\n",
       "      <td>10.0821</td>\n",
       "      <td>42.9544</td>\n",
       "      <td>79.4778</td>\n",
       "      <td>20.4542</td>\n",
       "      <td>88.5511</td>\n",
       "      <td>70.1653</td>\n",
       "      <td>178.7903</td>\n",
       "      <td>0.0</td>\n",
       "      <td>9.6490</td>\n",
       "      <td>...</td>\n",
       "      <td>6.3679</td>\n",
       "      <td>183.6078</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>7.0177</td>\n",
       "      <td>153.4365</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>22.9161</td>\n",
       "      <td>76.4800</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>20170503</th>\n",
       "      <td>6.7697</td>\n",
       "      <td>9.9035</td>\n",
       "      <td>42.6314</td>\n",
       "      <td>78.8332</td>\n",
       "      <td>20.1893</td>\n",
       "      <td>88.3302</td>\n",
       "      <td>69.4123</td>\n",
       "      <td>176.8719</td>\n",
       "      <td>0.0</td>\n",
       "      <td>9.5060</td>\n",
       "      <td>...</td>\n",
       "      <td>6.3143</td>\n",
       "      <td>180.7143</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>6.9472</td>\n",
       "      <td>155.2097</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>22.9674</td>\n",
       "      <td>75.6340</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>20170504</th>\n",
       "      <td>6.6405</td>\n",
       "      <td>9.9876</td>\n",
       "      <td>42.4161</td>\n",
       "      <td>78.6490</td>\n",
       "      <td>20.2187</td>\n",
       "      <td>86.9501</td>\n",
       "      <td>68.1117</td>\n",
       "      <td>175.1454</td>\n",
       "      <td>0.0</td>\n",
       "      <td>9.5298</td>\n",
       "      <td>...</td>\n",
       "      <td>6.3143</td>\n",
       "      <td>178.0838</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>6.9002</td>\n",
       "      <td>150.8288</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>22.8647</td>\n",
       "      <td>73.7727</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>20170505</th>\n",
       "      <td>6.5570</td>\n",
       "      <td>9.9193</td>\n",
       "      <td>40.5860</td>\n",
       "      <td>76.0703</td>\n",
       "      <td>20.0127</td>\n",
       "      <td>83.9137</td>\n",
       "      <td>67.6325</td>\n",
       "      <td>174.3781</td>\n",
       "      <td>0.0</td>\n",
       "      <td>9.3869</td>\n",
       "      <td>...</td>\n",
       "      <td>6.3322</td>\n",
       "      <td>174.4011</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>6.8649</td>\n",
       "      <td>147.1780</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>23.1319</td>\n",
       "      <td>72.2499</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>20170508</th>\n",
       "      <td>6.5114</td>\n",
       "      <td>9.6988</td>\n",
       "      <td>39.3479</td>\n",
       "      <td>74.2284</td>\n",
       "      <td>19.6007</td>\n",
       "      <td>82.9752</td>\n",
       "      <td>67.9063</td>\n",
       "      <td>175.3372</td>\n",
       "      <td>0.0</td>\n",
       "      <td>9.3273</td>\n",
       "      <td>...</td>\n",
       "      <td>6.3858</td>\n",
       "      <td>168.8771</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>6.9002</td>\n",
       "      <td>146.7608</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>22.7311</td>\n",
       "      <td>72.5883</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>5 rows × 330 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "symbol      000001.SZ  000002.SZ  000008.SZ  000009.SZ  000027.SZ  000039.SZ  \\\n",
       "trade_date                                                                     \n",
       "20170502       6.7925    10.0821    42.9544    79.4778    20.4542    88.5511   \n",
       "20170503       6.7697     9.9035    42.6314    78.8332    20.1893    88.3302   \n",
       "20170504       6.6405     9.9876    42.4161    78.6490    20.2187    86.9501   \n",
       "20170505       6.5570     9.9193    40.5860    76.0703    20.0127    83.9137   \n",
       "20170508       6.5114     9.6988    39.3479    74.2284    19.6007    82.9752   \n",
       "\n",
       "symbol      000060.SZ  000061.SZ  000063.SZ  000069.SZ    ...      601988.SH  \\\n",
       "trade_date                                                ...                  \n",
       "20170502      70.1653   178.7903        0.0     9.6490    ...         6.3679   \n",
       "20170503      69.4123   176.8719        0.0     9.5060    ...         6.3143   \n",
       "20170504      68.1117   175.1454        0.0     9.5298    ...         6.3143   \n",
       "20170505      67.6325   174.3781        0.0     9.3869    ...         6.3322   \n",
       "20170508      67.9063   175.3372        0.0     9.3273    ...         6.3858   \n",
       "\n",
       "symbol      601989.SH  601992.SH  601997.SH  601998.SH  603000.SH  603160.SH  \\\n",
       "trade_date                                                                     \n",
       "20170502     183.6078        NaN        NaN     7.0177   153.4365        NaN   \n",
       "20170503     180.7143        NaN        NaN     6.9472   155.2097        NaN   \n",
       "20170504     178.0838        NaN        NaN     6.9002   150.8288        NaN   \n",
       "20170505     174.4011        NaN        NaN     6.8649   147.1780        NaN   \n",
       "20170508     168.8771        NaN        NaN     6.9002   146.7608        NaN   \n",
       "\n",
       "symbol      603858.SH  603885.SH  603993.SH  \n",
       "trade_date                                   \n",
       "20170502          NaN    22.9161    76.4800  \n",
       "20170503          NaN    22.9674    75.6340  \n",
       "20170504          NaN    22.8647    73.7727  \n",
       "20170505          NaN    23.1319    72.2499  \n",
       "20170508          NaN    22.7311    72.5883  \n",
       "\n",
       "[5 rows x 330 columns]"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from jaqs_fxdayu.research.signaldigger.process import winsorize\n",
    "\n",
    "winsorize(factor_df = dv.get_ts(\"pe\"), \n",
    "          alpha=0.05,\n",
    "          index_member = dv.get_ts(\"index_member\")).head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## mad\n",
    "- ` jaqs_fxdayu.research.signaldigger.process.mad(factor_df, index_member=None) `\n",
    "\n",
    "**简要描述：**\n",
    "\n",
    "- 横截面去极值\n",
    "\n",
    "**参数:**\n",
    "\n",
    "|字段|必选|类型|说明|\n",
    "|:----    |:---|:----- |-----   |\n",
    "|factor_df |是|pandas.DataFrame |日期为索引,证券品种为columns的二维因子表格|\n",
    "|index_member |否|pandas.DataFrame of bool |是否是指数成分股。日期为索引,证券品种为columns的二维bool值表格,True代表该品种在该日期下属于指数成分股。传入该参数,则进行去极值所纳入的样本只有每期横截面上属于对应指数成分股的股票，默认为空|\n",
    "\n",
    "**返回:**\n",
    "\n",
    "去极值后的因子\n",
    "\n",
    "**示例：**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style>\n",
       "    .dataframe thead tr:only-child th {\n",
       "        text-align: right;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: left;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th>symbol</th>\n",
       "      <th>000001.SZ</th>\n",
       "      <th>000002.SZ</th>\n",
       "      <th>000008.SZ</th>\n",
       "      <th>000009.SZ</th>\n",
       "      <th>000027.SZ</th>\n",
       "      <th>000039.SZ</th>\n",
       "      <th>000060.SZ</th>\n",
       "      <th>000061.SZ</th>\n",
       "      <th>000063.SZ</th>\n",
       "      <th>000069.SZ</th>\n",
       "      <th>...</th>\n",
       "      <th>601988.SH</th>\n",
       "      <th>601989.SH</th>\n",
       "      <th>601992.SH</th>\n",
       "      <th>601997.SH</th>\n",
       "      <th>601998.SH</th>\n",
       "      <th>603000.SH</th>\n",
       "      <th>603160.SH</th>\n",
       "      <th>603858.SH</th>\n",
       "      <th>603885.SH</th>\n",
       "      <th>603993.SH</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>trade_date</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>20170502</th>\n",
       "      <td>6.7925</td>\n",
       "      <td>10.0821</td>\n",
       "      <td>42.9544</td>\n",
       "      <td>79.4778</td>\n",
       "      <td>20.4542</td>\n",
       "      <td>88.5511</td>\n",
       "      <td>70.1653</td>\n",
       "      <td>91.92400</td>\n",
       "      <td>0.0</td>\n",
       "      <td>9.6490</td>\n",
       "      <td>...</td>\n",
       "      <td>6.3679</td>\n",
       "      <td>91.92400</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>7.0177</td>\n",
       "      <td>91.92400</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>22.9161</td>\n",
       "      <td>76.4800</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>20170503</th>\n",
       "      <td>6.7697</td>\n",
       "      <td>9.9035</td>\n",
       "      <td>42.6314</td>\n",
       "      <td>78.8332</td>\n",
       "      <td>20.1893</td>\n",
       "      <td>88.3302</td>\n",
       "      <td>69.4123</td>\n",
       "      <td>91.87230</td>\n",
       "      <td>0.0</td>\n",
       "      <td>9.5060</td>\n",
       "      <td>...</td>\n",
       "      <td>6.3143</td>\n",
       "      <td>91.87230</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>6.9472</td>\n",
       "      <td>91.87230</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>22.9674</td>\n",
       "      <td>75.6340</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>20170504</th>\n",
       "      <td>6.6405</td>\n",
       "      <td>9.9876</td>\n",
       "      <td>42.4161</td>\n",
       "      <td>78.6490</td>\n",
       "      <td>20.2187</td>\n",
       "      <td>86.9501</td>\n",
       "      <td>68.1117</td>\n",
       "      <td>92.15105</td>\n",
       "      <td>0.0</td>\n",
       "      <td>9.5298</td>\n",
       "      <td>...</td>\n",
       "      <td>6.3143</td>\n",
       "      <td>92.15105</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>6.9002</td>\n",
       "      <td>92.15105</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>22.8647</td>\n",
       "      <td>73.7727</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>20170505</th>\n",
       "      <td>6.5570</td>\n",
       "      <td>9.9193</td>\n",
       "      <td>40.5860</td>\n",
       "      <td>76.0703</td>\n",
       "      <td>20.0127</td>\n",
       "      <td>83.9137</td>\n",
       "      <td>67.6325</td>\n",
       "      <td>86.81125</td>\n",
       "      <td>0.0</td>\n",
       "      <td>9.3869</td>\n",
       "      <td>...</td>\n",
       "      <td>6.3322</td>\n",
       "      <td>86.81125</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>6.8649</td>\n",
       "      <td>86.81125</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>23.1319</td>\n",
       "      <td>72.2499</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>20170508</th>\n",
       "      <td>6.5114</td>\n",
       "      <td>9.6988</td>\n",
       "      <td>39.3479</td>\n",
       "      <td>74.2284</td>\n",
       "      <td>19.6007</td>\n",
       "      <td>82.9752</td>\n",
       "      <td>67.9063</td>\n",
       "      <td>86.30405</td>\n",
       "      <td>0.0</td>\n",
       "      <td>9.3273</td>\n",
       "      <td>...</td>\n",
       "      <td>6.3858</td>\n",
       "      <td>86.30405</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>6.9002</td>\n",
       "      <td>86.30405</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>22.7311</td>\n",
       "      <td>72.5883</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>5 rows × 330 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "symbol      000001.SZ  000002.SZ  000008.SZ  000009.SZ  000027.SZ  000039.SZ  \\\n",
       "trade_date                                                                     \n",
       "20170502       6.7925    10.0821    42.9544    79.4778    20.4542    88.5511   \n",
       "20170503       6.7697     9.9035    42.6314    78.8332    20.1893    88.3302   \n",
       "20170504       6.6405     9.9876    42.4161    78.6490    20.2187    86.9501   \n",
       "20170505       6.5570     9.9193    40.5860    76.0703    20.0127    83.9137   \n",
       "20170508       6.5114     9.6988    39.3479    74.2284    19.6007    82.9752   \n",
       "\n",
       "symbol      000060.SZ  000061.SZ  000063.SZ  000069.SZ    ...      601988.SH  \\\n",
       "trade_date                                                ...                  \n",
       "20170502      70.1653   91.92400        0.0     9.6490    ...         6.3679   \n",
       "20170503      69.4123   91.87230        0.0     9.5060    ...         6.3143   \n",
       "20170504      68.1117   92.15105        0.0     9.5298    ...         6.3143   \n",
       "20170505      67.6325   86.81125        0.0     9.3869    ...         6.3322   \n",
       "20170508      67.9063   86.30405        0.0     9.3273    ...         6.3858   \n",
       "\n",
       "symbol      601989.SH  601992.SH  601997.SH  601998.SH  603000.SH  603160.SH  \\\n",
       "trade_date                                                                     \n",
       "20170502     91.92400        NaN        NaN     7.0177   91.92400        NaN   \n",
       "20170503     91.87230        NaN        NaN     6.9472   91.87230        NaN   \n",
       "20170504     92.15105        NaN        NaN     6.9002   92.15105        NaN   \n",
       "20170505     86.81125        NaN        NaN     6.8649   86.81125        NaN   \n",
       "20170508     86.30405        NaN        NaN     6.9002   86.30405        NaN   \n",
       "\n",
       "symbol      603858.SH  603885.SH  603993.SH  \n",
       "trade_date                                   \n",
       "20170502          NaN    22.9161    76.4800  \n",
       "20170503          NaN    22.9674    75.6340  \n",
       "20170504          NaN    22.8647    73.7727  \n",
       "20170505          NaN    23.1319    72.2499  \n",
       "20170508          NaN    22.7311    72.5883  \n",
       "\n",
       "[5 rows x 330 columns]"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from jaqs_fxdayu.research.signaldigger.process import mad\n",
    "\n",
    "mad(factor_df = dv.get_ts(\"pe\"), \n",
    "    index_member = dv.get_ts(\"index_member\")).head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## rank_standardize\n",
    "- ` jaqs_fxdayu.research.signaldigger.process.rank_standardize(factor_df, index_member=None) `\n",
    "\n",
    "**简要描述：**\n",
    "\n",
    "- 排序标准化。将因子处理成横截面上的排序值（升序），并处理到0-1之间——仅保留原因子的顺序特征，剔除分布特征\n",
    "\n",
    "**参数:**\n",
    "\n",
    "|字段|必选|类型|说明|\n",
    "|:----    |:---|:----- |-----   |\n",
    "|factor_df |是|pandas.DataFrame |日期为索引,证券品种为columns的二维因子表格|\n",
    "|index_member |否|pandas.DataFrame of bool |是否是指数成分股。日期为索引,证券品种为columns的二维bool值表格,True代表该品种在该日期下属于指数成分股。传入该参数,则进行排序标准化所纳入的样本只有每期横截面上属于对应指数成分股的股票，默认为空|\n",
    "\n",
    "**返回:**\n",
    "\n",
    "排序标准化后的因子\n",
    "\n",
    "**示例：**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style>\n",
       "    .dataframe thead tr:only-child th {\n",
       "        text-align: right;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: left;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th>symbol</th>\n",
       "      <th>000001.SZ</th>\n",
       "      <th>000002.SZ</th>\n",
       "      <th>000008.SZ</th>\n",
       "      <th>000009.SZ</th>\n",
       "      <th>000027.SZ</th>\n",
       "      <th>000039.SZ</th>\n",
       "      <th>000060.SZ</th>\n",
       "      <th>000061.SZ</th>\n",
       "      <th>000063.SZ</th>\n",
       "      <th>000069.SZ</th>\n",
       "      <th>...</th>\n",
       "      <th>601988.SH</th>\n",
       "      <th>601989.SH</th>\n",
       "      <th>601992.SH</th>\n",
       "      <th>601997.SH</th>\n",
       "      <th>601998.SH</th>\n",
       "      <th>603000.SH</th>\n",
       "      <th>603160.SH</th>\n",
       "      <th>603858.SH</th>\n",
       "      <th>603885.SH</th>\n",
       "      <th>603993.SH</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>trade_date</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>20170502</th>\n",
       "      <td>0.063545</td>\n",
       "      <td>0.117057</td>\n",
       "      <td>0.722408</td>\n",
       "      <td>0.886288</td>\n",
       "      <td>0.361204</td>\n",
       "      <td>0.90301</td>\n",
       "      <td>0.862876</td>\n",
       "      <td>0.966555</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.107023</td>\n",
       "      <td>...</td>\n",
       "      <td>0.053512</td>\n",
       "      <td>0.969900</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>0.070234</td>\n",
       "      <td>0.943144</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>0.408027</td>\n",
       "      <td>0.876254</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>20170503</th>\n",
       "      <td>0.063545</td>\n",
       "      <td>0.113712</td>\n",
       "      <td>0.722408</td>\n",
       "      <td>0.886288</td>\n",
       "      <td>0.354515</td>\n",
       "      <td>0.90301</td>\n",
       "      <td>0.859532</td>\n",
       "      <td>0.966555</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.100334</td>\n",
       "      <td>...</td>\n",
       "      <td>0.053512</td>\n",
       "      <td>0.969900</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>0.066890</td>\n",
       "      <td>0.939799</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>0.408027</td>\n",
       "      <td>0.872910</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>20170504</th>\n",
       "      <td>0.063545</td>\n",
       "      <td>0.113712</td>\n",
       "      <td>0.725753</td>\n",
       "      <td>0.886288</td>\n",
       "      <td>0.357860</td>\n",
       "      <td>0.90301</td>\n",
       "      <td>0.852843</td>\n",
       "      <td>0.963211</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.100334</td>\n",
       "      <td>...</td>\n",
       "      <td>0.053512</td>\n",
       "      <td>0.969900</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>0.066890</td>\n",
       "      <td>0.939799</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>0.408027</td>\n",
       "      <td>0.872910</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>20170505</th>\n",
       "      <td>0.063545</td>\n",
       "      <td>0.113712</td>\n",
       "      <td>0.712375</td>\n",
       "      <td>0.882943</td>\n",
       "      <td>0.351171</td>\n",
       "      <td>0.90301</td>\n",
       "      <td>0.859532</td>\n",
       "      <td>0.963211</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.100334</td>\n",
       "      <td>...</td>\n",
       "      <td>0.053512</td>\n",
       "      <td>0.966555</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>0.070234</td>\n",
       "      <td>0.943144</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>0.424749</td>\n",
       "      <td>0.872910</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>20170508</th>\n",
       "      <td>0.060201</td>\n",
       "      <td>0.103679</td>\n",
       "      <td>0.719064</td>\n",
       "      <td>0.882943</td>\n",
       "      <td>0.331104</td>\n",
       "      <td>0.90301</td>\n",
       "      <td>0.862876</td>\n",
       "      <td>0.969900</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.096990</td>\n",
       "      <td>...</td>\n",
       "      <td>0.053512</td>\n",
       "      <td>0.963211</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>0.070234</td>\n",
       "      <td>0.946488</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>0.421405</td>\n",
       "      <td>0.879599</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>5 rows × 330 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "symbol      000001.SZ  000002.SZ  000008.SZ  000009.SZ  000027.SZ  000039.SZ  \\\n",
       "trade_date                                                                     \n",
       "20170502     0.063545   0.117057   0.722408   0.886288   0.361204    0.90301   \n",
       "20170503     0.063545   0.113712   0.722408   0.886288   0.354515    0.90301   \n",
       "20170504     0.063545   0.113712   0.725753   0.886288   0.357860    0.90301   \n",
       "20170505     0.063545   0.113712   0.712375   0.882943   0.351171    0.90301   \n",
       "20170508     0.060201   0.103679   0.719064   0.882943   0.331104    0.90301   \n",
       "\n",
       "symbol      000060.SZ  000061.SZ  000063.SZ  000069.SZ    ...      601988.SH  \\\n",
       "trade_date                                                ...                  \n",
       "20170502     0.862876   0.966555        0.0   0.107023    ...       0.053512   \n",
       "20170503     0.859532   0.966555        0.0   0.100334    ...       0.053512   \n",
       "20170504     0.852843   0.963211        0.0   0.100334    ...       0.053512   \n",
       "20170505     0.859532   0.963211        0.0   0.100334    ...       0.053512   \n",
       "20170508     0.862876   0.969900        0.0   0.096990    ...       0.053512   \n",
       "\n",
       "symbol      601989.SH  601992.SH  601997.SH  601998.SH  603000.SH  603160.SH  \\\n",
       "trade_date                                                                     \n",
       "20170502     0.969900        NaN        NaN   0.070234   0.943144        NaN   \n",
       "20170503     0.969900        NaN        NaN   0.066890   0.939799        NaN   \n",
       "20170504     0.969900        NaN        NaN   0.066890   0.939799        NaN   \n",
       "20170505     0.966555        NaN        NaN   0.070234   0.943144        NaN   \n",
       "20170508     0.963211        NaN        NaN   0.070234   0.946488        NaN   \n",
       "\n",
       "symbol      603858.SH  603885.SH  603993.SH  \n",
       "trade_date                                   \n",
       "20170502          NaN   0.408027   0.876254  \n",
       "20170503          NaN   0.408027   0.872910  \n",
       "20170504          NaN   0.408027   0.872910  \n",
       "20170505          NaN   0.424749   0.872910  \n",
       "20170508          NaN   0.421405   0.879599  \n",
       "\n",
       "[5 rows x 330 columns]"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from jaqs_fxdayu.research.signaldigger.process import rank_standardize\n",
    "\n",
    "rank_standardize(factor_df = dv.get_ts(\"pe\"), \n",
    "                 index_member = dv.get_ts(\"index_member\")).head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## get_disturbed_factor\n",
    "- ` jaqs_fxdayu.research.signaldigger.process.rank_standardizeget_disturbed_factor(factor_df) `\n",
    "\n",
    "**简要描述：**\n",
    "\n",
    "- 将因子值加一个极小的扰动项,用于对quantile分组做区分\n",
    "\n",
    "**参数:**\n",
    "\n",
    "|字段|必选|类型|说明|\n",
    "|:----    |:---|:----- |-----   |\n",
    "|factor_df |是|pandas.DataFrame |日期为索引,证券品种为columns的二维因子表格|\n",
    "\n",
    "**返回:**\n",
    "\n",
    "加扰动项后的因子\n",
    "\n",
    "**示例：**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style>\n",
       "    .dataframe thead tr:only-child th {\n",
       "        text-align: right;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: left;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th>symbol</th>\n",
       "      <th>000001.SZ</th>\n",
       "      <th>000002.SZ</th>\n",
       "      <th>000008.SZ</th>\n",
       "      <th>000009.SZ</th>\n",
       "      <th>000027.SZ</th>\n",
       "      <th>000039.SZ</th>\n",
       "      <th>000060.SZ</th>\n",
       "      <th>000061.SZ</th>\n",
       "      <th>000063.SZ</th>\n",
       "      <th>000069.SZ</th>\n",
       "      <th>...</th>\n",
       "      <th>601988.SH</th>\n",
       "      <th>601989.SH</th>\n",
       "      <th>601992.SH</th>\n",
       "      <th>601997.SH</th>\n",
       "      <th>601998.SH</th>\n",
       "      <th>603000.SH</th>\n",
       "      <th>603160.SH</th>\n",
       "      <th>603858.SH</th>\n",
       "      <th>603885.SH</th>\n",
       "      <th>603993.SH</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>trade_date</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>20170502</th>\n",
       "      <td>6.7925</td>\n",
       "      <td>10.0821</td>\n",
       "      <td>42.9544</td>\n",
       "      <td>79.4778</td>\n",
       "      <td>20.4542</td>\n",
       "      <td>88.5511</td>\n",
       "      <td>70.1653</td>\n",
       "      <td>178.7903</td>\n",
       "      <td>3.437688e-10</td>\n",
       "      <td>9.6490</td>\n",
       "      <td>...</td>\n",
       "      <td>6.3679</td>\n",
       "      <td>183.6078</td>\n",
       "      <td>32.7886</td>\n",
       "      <td>9.8565</td>\n",
       "      <td>7.0177</td>\n",
       "      <td>153.4365</td>\n",
       "      <td>50.8349</td>\n",
       "      <td>31.0157</td>\n",
       "      <td>22.9161</td>\n",
       "      <td>76.4800</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>20170503</th>\n",
       "      <td>6.7697</td>\n",
       "      <td>9.9035</td>\n",
       "      <td>42.6314</td>\n",
       "      <td>78.8332</td>\n",
       "      <td>20.1893</td>\n",
       "      <td>88.3302</td>\n",
       "      <td>69.4123</td>\n",
       "      <td>176.8719</td>\n",
       "      <td>4.412786e-10</td>\n",
       "      <td>9.5060</td>\n",
       "      <td>...</td>\n",
       "      <td>6.3143</td>\n",
       "      <td>180.7143</td>\n",
       "      <td>30.2450</td>\n",
       "      <td>9.8817</td>\n",
       "      <td>6.9472</td>\n",
       "      <td>155.2097</td>\n",
       "      <td>50.7259</td>\n",
       "      <td>31.0311</td>\n",
       "      <td>22.9674</td>\n",
       "      <td>75.6340</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>20170504</th>\n",
       "      <td>6.6405</td>\n",
       "      <td>9.9876</td>\n",
       "      <td>42.4161</td>\n",
       "      <td>78.6490</td>\n",
       "      <td>20.2187</td>\n",
       "      <td>86.9501</td>\n",
       "      <td>68.1117</td>\n",
       "      <td>175.1454</td>\n",
       "      <td>4.559244e-10</td>\n",
       "      <td>9.5298</td>\n",
       "      <td>...</td>\n",
       "      <td>6.3143</td>\n",
       "      <td>178.0838</td>\n",
       "      <td>31.4771</td>\n",
       "      <td>9.8188</td>\n",
       "      <td>6.9002</td>\n",
       "      <td>150.8288</td>\n",
       "      <td>50.3727</td>\n",
       "      <td>30.6805</td>\n",
       "      <td>22.8647</td>\n",
       "      <td>73.7727</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>20170505</th>\n",
       "      <td>6.5570</td>\n",
       "      <td>9.9193</td>\n",
       "      <td>40.5860</td>\n",
       "      <td>76.0703</td>\n",
       "      <td>20.0127</td>\n",
       "      <td>83.9137</td>\n",
       "      <td>67.6325</td>\n",
       "      <td>174.3781</td>\n",
       "      <td>6.587699e-10</td>\n",
       "      <td>9.3869</td>\n",
       "      <td>...</td>\n",
       "      <td>6.3322</td>\n",
       "      <td>174.4011</td>\n",
       "      <td>30.8809</td>\n",
       "      <td>9.5609</td>\n",
       "      <td>6.8649</td>\n",
       "      <td>147.1780</td>\n",
       "      <td>49.3963</td>\n",
       "      <td>30.2527</td>\n",
       "      <td>23.1319</td>\n",
       "      <td>72.2499</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>20170508</th>\n",
       "      <td>6.5114</td>\n",
       "      <td>9.6988</td>\n",
       "      <td>39.3479</td>\n",
       "      <td>74.2284</td>\n",
       "      <td>19.6007</td>\n",
       "      <td>82.9752</td>\n",
       "      <td>67.9063</td>\n",
       "      <td>175.3372</td>\n",
       "      <td>6.254412e-10</td>\n",
       "      <td>9.3273</td>\n",
       "      <td>...</td>\n",
       "      <td>6.3858</td>\n",
       "      <td>168.8771</td>\n",
       "      <td>27.9399</td>\n",
       "      <td>9.3282</td>\n",
       "      <td>6.9002</td>\n",
       "      <td>146.7608</td>\n",
       "      <td>50.3779</td>\n",
       "      <td>29.5167</td>\n",
       "      <td>22.7311</td>\n",
       "      <td>72.5883</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>5 rows × 330 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "symbol      000001.SZ  000002.SZ  000008.SZ  000009.SZ  000027.SZ  000039.SZ  \\\n",
       "trade_date                                                                     \n",
       "20170502       6.7925    10.0821    42.9544    79.4778    20.4542    88.5511   \n",
       "20170503       6.7697     9.9035    42.6314    78.8332    20.1893    88.3302   \n",
       "20170504       6.6405     9.9876    42.4161    78.6490    20.2187    86.9501   \n",
       "20170505       6.5570     9.9193    40.5860    76.0703    20.0127    83.9137   \n",
       "20170508       6.5114     9.6988    39.3479    74.2284    19.6007    82.9752   \n",
       "\n",
       "symbol      000060.SZ  000061.SZ     000063.SZ  000069.SZ    ...      \\\n",
       "trade_date                                                   ...       \n",
       "20170502      70.1653   178.7903  3.437688e-10     9.6490    ...       \n",
       "20170503      69.4123   176.8719  4.412786e-10     9.5060    ...       \n",
       "20170504      68.1117   175.1454  4.559244e-10     9.5298    ...       \n",
       "20170505      67.6325   174.3781  6.587699e-10     9.3869    ...       \n",
       "20170508      67.9063   175.3372  6.254412e-10     9.3273    ...       \n",
       "\n",
       "symbol      601988.SH  601989.SH  601992.SH  601997.SH  601998.SH  603000.SH  \\\n",
       "trade_date                                                                     \n",
       "20170502       6.3679   183.6078    32.7886     9.8565     7.0177   153.4365   \n",
       "20170503       6.3143   180.7143    30.2450     9.8817     6.9472   155.2097   \n",
       "20170504       6.3143   178.0838    31.4771     9.8188     6.9002   150.8288   \n",
       "20170505       6.3322   174.4011    30.8809     9.5609     6.8649   147.1780   \n",
       "20170508       6.3858   168.8771    27.9399     9.3282     6.9002   146.7608   \n",
       "\n",
       "symbol      603160.SH  603858.SH  603885.SH  603993.SH  \n",
       "trade_date                                              \n",
       "20170502      50.8349    31.0157    22.9161    76.4800  \n",
       "20170503      50.7259    31.0311    22.9674    75.6340  \n",
       "20170504      50.3727    30.6805    22.8647    73.7727  \n",
       "20170505      49.3963    30.2527    23.1319    72.2499  \n",
       "20170508      50.3779    29.5167    22.7311    72.5883  \n",
       "\n",
       "[5 rows x 330 columns]"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from jaqs_fxdayu.research.signaldigger.process import get_disturbed_factor\n",
    "\n",
    "get_disturbed_factor(factor_df = dv.get_ts(\"pe\")).head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## neutralize\n",
    "- ` jaqs_fxdayu.research.signaldigger.process.neutralize(factor_df,group,float_mv=None,index_member=None) `\n",
    "\n",
    "**简要描述：**\n",
    "\n",
    "- 对因子做行业、市值中性化\n",
    "\n",
    "**参数:**\n",
    "\n",
    "|字段|必选|类型|说明|\n",
    "|:----    |:---|:----- |-----   |\n",
    "|factor_df |是|pandas.DataFrame |因子。日期为索引,证券品种为columns的二维表格|\n",
    "|group |是|pandas.DataFrame |行业分类（也可以是其他分组方式）。日期为索引,证券品种为columns的二维表格,对应每一个品种在某期所属的分类|\n",
    "|float_mv |否|pandas.DataFrame |流通市值。日期为索引,证券品种为columns的二维表格。默认为空,为空时不进行市值中性化处理|\n",
    "|index_member |否|pandas.DataFrame of bool |是否是指数成分股。日期为索引,证券品种为columns的二维bool值表格,True代表该品种在该日期下属于指数成分股。传入该参数,则进行行业、市值中性化所纳入的样本只有每期横截面上属于对应指数成分股的股票，默认为空|\n",
    "\n",
    "**返回:**\n",
    "\n",
    "行业、市值中性化后的因子\n",
    "\n",
    "**示例：**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style>\n",
       "    .dataframe thead tr:only-child th {\n",
       "        text-align: right;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: left;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th>symbol</th>\n",
       "      <th>000001.SZ</th>\n",
       "      <th>000002.SZ</th>\n",
       "      <th>000008.SZ</th>\n",
       "      <th>000009.SZ</th>\n",
       "      <th>000027.SZ</th>\n",
       "      <th>000039.SZ</th>\n",
       "      <th>000060.SZ</th>\n",
       "      <th>000061.SZ</th>\n",
       "      <th>000063.SZ</th>\n",
       "      <th>000069.SZ</th>\n",
       "      <th>...</th>\n",
       "      <th>601988.SH</th>\n",
       "      <th>601989.SH</th>\n",
       "      <th>601992.SH</th>\n",
       "      <th>601997.SH</th>\n",
       "      <th>601998.SH</th>\n",
       "      <th>603000.SH</th>\n",
       "      <th>603160.SH</th>\n",
       "      <th>603858.SH</th>\n",
       "      <th>603885.SH</th>\n",
       "      <th>603993.SH</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>trade_date</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>20170502</th>\n",
       "      <td>-2.662629</td>\n",
       "      <td>-7.782230</td>\n",
       "      <td>-27.98201</td>\n",
       "      <td>-38.33350</td>\n",
       "      <td>-4.083109</td>\n",
       "      <td>17.61469</td>\n",
       "      <td>-83.013425</td>\n",
       "      <td>107.385838</td>\n",
       "      <td>-168.217857</td>\n",
       "      <td>-8.215330</td>\n",
       "      <td>...</td>\n",
       "      <td>-3.087229</td>\n",
       "      <td>26.912500</td>\n",
       "      <td>9.87725</td>\n",
       "      <td>0.401371</td>\n",
       "      <td>-2.437429</td>\n",
       "      <td>108.584833</td>\n",
       "      <td>3.357346</td>\n",
       "      <td>-55.150405</td>\n",
       "      <td>-6.266428</td>\n",
       "      <td>-76.698725</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>20170503</th>\n",
       "      <td>-2.682662</td>\n",
       "      <td>-7.829960</td>\n",
       "      <td>-28.76077</td>\n",
       "      <td>-39.50720</td>\n",
       "      <td>-3.544909</td>\n",
       "      <td>16.93803</td>\n",
       "      <td>-83.589442</td>\n",
       "      <td>105.819463</td>\n",
       "      <td>-168.313357</td>\n",
       "      <td>-8.227460</td>\n",
       "      <td>...</td>\n",
       "      <td>-3.138062</td>\n",
       "      <td>24.588600</td>\n",
       "      <td>8.60545</td>\n",
       "      <td>0.429338</td>\n",
       "      <td>-2.505162</td>\n",
       "      <td>110.440158</td>\n",
       "      <td>3.330400</td>\n",
       "      <td>-55.949523</td>\n",
       "      <td>-6.084489</td>\n",
       "      <td>-77.367742</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>20170504</th>\n",
       "      <td>-2.815043</td>\n",
       "      <td>-7.733890</td>\n",
       "      <td>-28.67189</td>\n",
       "      <td>-38.74790</td>\n",
       "      <td>-4.016945</td>\n",
       "      <td>15.86211</td>\n",
       "      <td>-82.429800</td>\n",
       "      <td>104.271487</td>\n",
       "      <td>-168.140586</td>\n",
       "      <td>-8.191690</td>\n",
       "      <td>...</td>\n",
       "      <td>-3.141243</td>\n",
       "      <td>26.910367</td>\n",
       "      <td>9.39235</td>\n",
       "      <td>0.363257</td>\n",
       "      <td>-2.555343</td>\n",
       "      <td>106.489883</td>\n",
       "      <td>3.025662</td>\n",
       "      <td>-55.572859</td>\n",
       "      <td>-6.116911</td>\n",
       "      <td>-76.768800</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>20170505</th>\n",
       "      <td>-2.762233</td>\n",
       "      <td>-7.653145</td>\n",
       "      <td>-28.89854</td>\n",
       "      <td>-37.39795</td>\n",
       "      <td>-3.835882</td>\n",
       "      <td>14.42916</td>\n",
       "      <td>-82.012883</td>\n",
       "      <td>103.912075</td>\n",
       "      <td>-167.959957</td>\n",
       "      <td>-8.185545</td>\n",
       "      <td>...</td>\n",
       "      <td>-2.987033</td>\n",
       "      <td>27.853900</td>\n",
       "      <td>9.18745</td>\n",
       "      <td>0.241667</td>\n",
       "      <td>-2.454333</td>\n",
       "      <td>103.302950</td>\n",
       "      <td>2.387592</td>\n",
       "      <td>-55.251945</td>\n",
       "      <td>-5.618411</td>\n",
       "      <td>-77.395483</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>20170508</th>\n",
       "      <td>-2.564538</td>\n",
       "      <td>-7.591140</td>\n",
       "      <td>-29.02696</td>\n",
       "      <td>-36.10530</td>\n",
       "      <td>-3.576855</td>\n",
       "      <td>14.60034</td>\n",
       "      <td>-80.613975</td>\n",
       "      <td>104.639975</td>\n",
       "      <td>-167.807429</td>\n",
       "      <td>-7.962640</td>\n",
       "      <td>...</td>\n",
       "      <td>-2.690138</td>\n",
       "      <td>30.356133</td>\n",
       "      <td>7.68590</td>\n",
       "      <td>0.252262</td>\n",
       "      <td>-2.175738</td>\n",
       "      <td>102.756587</td>\n",
       "      <td>4.286408</td>\n",
       "      <td>-54.397259</td>\n",
       "      <td>-5.803628</td>\n",
       "      <td>-75.931975</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>5 rows × 330 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "symbol      000001.SZ  000002.SZ  000008.SZ  000009.SZ  000027.SZ  000039.SZ  \\\n",
       "trade_date                                                                     \n",
       "20170502    -2.662629  -7.782230  -27.98201  -38.33350  -4.083109   17.61469   \n",
       "20170503    -2.682662  -7.829960  -28.76077  -39.50720  -3.544909   16.93803   \n",
       "20170504    -2.815043  -7.733890  -28.67189  -38.74790  -4.016945   15.86211   \n",
       "20170505    -2.762233  -7.653145  -28.89854  -37.39795  -3.835882   14.42916   \n",
       "20170508    -2.564538  -7.591140  -29.02696  -36.10530  -3.576855   14.60034   \n",
       "\n",
       "symbol      000060.SZ   000061.SZ   000063.SZ  000069.SZ    ...      \\\n",
       "trade_date                                                  ...       \n",
       "20170502   -83.013425  107.385838 -168.217857  -8.215330    ...       \n",
       "20170503   -83.589442  105.819463 -168.313357  -8.227460    ...       \n",
       "20170504   -82.429800  104.271487 -168.140586  -8.191690    ...       \n",
       "20170505   -82.012883  103.912075 -167.959957  -8.185545    ...       \n",
       "20170508   -80.613975  104.639975 -167.807429  -7.962640    ...       \n",
       "\n",
       "symbol      601988.SH  601989.SH  601992.SH  601997.SH  601998.SH   603000.SH  \\\n",
       "trade_date                                                                      \n",
       "20170502    -3.087229  26.912500    9.87725   0.401371  -2.437429  108.584833   \n",
       "20170503    -3.138062  24.588600    8.60545   0.429338  -2.505162  110.440158   \n",
       "20170504    -3.141243  26.910367    9.39235   0.363257  -2.555343  106.489883   \n",
       "20170505    -2.987033  27.853900    9.18745   0.241667  -2.454333  103.302950   \n",
       "20170508    -2.690138  30.356133    7.68590   0.252262  -2.175738  102.756587   \n",
       "\n",
       "symbol      603160.SH  603858.SH  603885.SH  603993.SH  \n",
       "trade_date                                              \n",
       "20170502     3.357346 -55.150405  -6.266428 -76.698725  \n",
       "20170503     3.330400 -55.949523  -6.084489 -77.367742  \n",
       "20170504     3.025662 -55.572859  -6.116911 -76.768800  \n",
       "20170505     2.387592 -55.251945  -5.618411 -77.395483  \n",
       "20170508     4.286408 -54.397259  -5.803628 -75.931975  \n",
       "\n",
       "[5 rows x 330 columns]"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from jaqs_fxdayu.research.signaldigger.process import neutralize\n",
    "\n",
    "neutralize(factor_df = dv.get_ts(\"pe\"),\n",
    "           group = dv.get_ts(\"sw1\")).head()"
   ]
  }
 ],
 "metadata": {
  "anaconda-cloud": {},
  "kernelspec": {
   "display_name": "Python [conda env:IIA]",
   "language": "python",
   "name": "conda-env-IIA-py"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.1"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 0
}
